1. Field of the Invention
This invention relates to the workflow employed in the creation of motion pictures and video and more specifically to the tasks encountered in digital intermediate processing in the post-production of motion pictures.
2. Description of the Related Art
Digital Intermediate (DI) Processing
While a small but growing number of motion pictures are being recorded and/or exhibited using digital cameras and/or digital projectors, most motion pictures are recorded and exhibited using film. The process of taking the originally recorded images and assembling them into the completed motion picture is known as post-production. Even though a motion picture may be recorded and exhibited using film, a large and growing number of motion pictures are employing digital processing techniques in post-production. This processing is known as digital intermediate (DI) processing. DI Processing facilitates performing many common post-production tasks by computer. These tasks are generically referred to here as “post-processing”. Examples of such tasks include: editing (deciding which images will be included and in what order), color correction, special effects, pan & scan, sub-titles, shot separation, re-framing, scene classification, artifact removal, resolution enhancement, noise reduction, sharpening, and verification. After DI processing, the images are recorded back to film to create the various prints involved in the mastering and distribution of motion pictures.
As shown in FIG. 1, original recorded data (step 12) consisting of a sequence of images 14 on film 16 is received at the post-production house. Each image 14 is scanned (step 17) to create a digital image 18. The sequence of digital images are stored (step 20) in mass storage. To maintain a high degree of faithfulness to the original film, scanning is often done at a high resolution. For example, digital images of sizes 2048×1080 (2K) or even 4096×2160 (4K) might result. Such large images can be quite cumbersome. Most computer monitors can not display images this large. Additionally, real time processing of such large images is often not possible.
The DI process creates a reduced resolution “proxy” of each image for post-processing. This is done by down sampling (step 22) the digital image 18 to a fixed resolution. Each down sampled version is then compressed (step 24) at a fixed quality level and the compressed proxies are stored (step 26). The selection of the resolution and quality level represents a tradeoff among processing power, storage capacity, display capability and the requirements of the various editing functions. When a post-processor (editor, colorist, etc.) requests a certain scene or cut (step 28), the corresponding compressed images are decompressed (step 30) to display the sequence of proxies 32. The proxies are distinct digital images, created from, but entirely separate from the digital versions of the original images. Storage of the compressed proxies is required in addition to storage of the digital versions of the original images.
The proxies are displayed on a computer workstation 34 and post-processed (step 36). As mentioned above, post-processing includes any processing that affects an image including but not limited to deciding which images are included in the motion picture and in what order they are to be viewed, color correction, etc. Post-processing generates one or more decision lists (step 38) that include a list of the operations that were carried out on each of the proxies. Typically, there would be a decision list for editing, another list for color correction and so on. Once editing, color correction etc., are completed on the proxies, the decision lists are applied to the digital images 18 so that the same operations are carried out on the digital versions of original images (step 40). The DI process produces an uncompressed digital master 42. The images in the digital master are recorded back to film to create the various prints involved in the mastering and distribution of motion pictures (step 44).
As mentioned above, the selection of the fixed resolution and fixed quality level for the “proxy” represents a tradeoff among processing power, storage capacity, display capability and the requirements of the various post-processing functions. The proxy is fixed and can not be optimized for any particular function or changing film content. As a result, the editor, colorist or special effects artists are limited in their capabilities to view the images, which in turn affects their efficiency and ultimately the quality of the digital master.
U.S. Pat. No. 5,577,191 describes a digital video editing and publishing system in which digital video data are compressed, intraframe-only, and stored. Selected frames of the compressed data are decompressed, edited and recompressed. The edited and compressed video sequence is decompressed and then recompressed using both intraframe and interframe compression. The parameters of the intraframe compression used in creating this digital master file are set such that the decompressed image is of a selected resolution and quality commensurate with the target publication medium. This approach combines the ability to decompress and edit individual frames by using intraframe compression only initially and the superior compression capability of intraframe and interframe compression. This approach does not utilize a proxy for editing. Consequently, if the target publication medium is of very high resolution the editing process may be very cumbersome.
Image Compression
Image compression is one of the key technologies fueling the expansion of applications such as DI processing that utilize digital imagery. Since the amount of data in images can be quite large, images are rarely transmitted or stored without compression. Image compression aims to represent an image with as few bits as possible while preserving the desired level of quality. The DI process follows downsampling by compression to create proxies that are sufficiently reduced to be processed in real-time and capable of display on editing workstations. Arguably, the most successful image compression standard has been the JPEG (Joint Photographic Experts Group) standard, which is often used to create proxies in the DI process.
JPEG2000 is the new international image compression standard (ISO/IEC 15444) and offers state-of-the-art compression performance for still imagery. JPEG2000 also offers a number of functionalities designed to specifically address the transmission and storage requirements of emerging imaging applications. In particular, JPEG2000 offers several mechanisms to provide for scalability and random access into compressed codestreams to reduce the amount of data to be transmitted during distribution of large digital images. To this end, the image data are compressed and stored in packets in the codestreams.
Low resolution versions of an image can be extracted and/or decompressed by accessing only the packets corresponding to low resolution subbands. For example, if the original image size is say 2048×1080, smaller versions of the image can be extracted at sizes of 1024×540, 512×270, etc. It might be said that the smaller versions are “zoomed out” or “overview” versions of the original. Each of the smaller images may contain the full spatial extent (field of view) of its corresponding original.
Reduced spatial extents (spatial regions) can be extracted and/or decompressed by accessing only the packets corresponding to the desired spatial region. This can be done at full resolution, or any reduced resolution as described in the previous paragraph. This feature allows the extraction of a “cropped” version of the original image, or a reduced resolution version of the cropped image.
Reduced quality images can be extracted by accessing only the packets corresponding to a number of initial quality layers. This can be done at full resolution, or any reduced resolution. Additionally, this can be done for the full spatial extent, or any spatial region as described in the previous paragraph. In fact, different qualities can be achieved at different spatial locations within the same image.
All of the features above can be used to extract a version of the image that is “reduced” in one way or another. In each case, only the packets relevant to the desired resolution, spatial extent, and/or quality need be accessed. Accessing other packets is not required. These features greatly decrease the amount of data to be transmitted in an image communication application
As described above, the JPEG2000 features were developed to facilitate transmission of large images. For this reason, it is being adopted for use in many applications. Specifically, JPEG2000 has been adopted by the medical imagery standards body known as DICOM for storage and distribution of medical imagery. Additionally, JPEG2000 is being used for distributing large overhead images as well as maps. These images and maps can be viewed in an interactive way to reveal different spatial regions at resolutions desired by the user. The library of congress is considering using JPEG2000 to allow users to browse archival imagery over the internet. Significantly, JPEG2000 has been selected for the distribution of Digital Motion Pictures. This selection was based largely on the fact that a single file can serve both high resolution, as well as medium resolution projectors. Specifically, the entire file may be suitable for a high resolution projector. Alternately, medium resolution imagery can be extracted (at the theater) for use by a medium resolution projector.